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ABSTRACT 



A conceptual approach for evaluating and selecting among 
alternative Electronic Data Processing (EDP) systems proposed 
to meet a set of EDP user needs has been developed by applying 
cost-effectiveness methods and techniques to the source selec- 
tion problem. The report provides a framework that allows the 
EDP system evaluator to combine the selected relevant system 
performance measures and the related cost elements to arrive 
at a rational defendable selection decision. 
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SECTION I 
INTRODUCTION 



The purpose of this technical report is to document a conceptual 
approach for evaluating and selecting among alternative, proposed 
Electronic Data Processing (EDP) systems designed to meet a set of EDP 
user needs. The proposed approach was developed by applying cost- 
effectiveness methods and techniques to the source selection problem. 
This work was accomplished under Project 8510. 

The selection of EDP systems has presented a problem to the 
decision-maker for many years since competition became established in 
the production of computing equipment . The development of computers 
has been characterized by a diversity of approaches, designs, and 
configurations . 

Solutions to this problem have been varied and cover a wide spec- 
trum all the way from essentially ignoring the technical differences 
to the carrying out of detailed studies utilizing sophisticated tools. 

This problem is not limited only to the EDP user. The manufac- 
turers themselves must make design decisions involving a trade-off 
between the cost of the equipment and its performance. For a partic- 
ular user the problem is basically simpler since he can confine his 
evaluation to how the EDP equipment proposed by the vendor satisfies 
his particular requirements. 

A significant amount of attention has been devoted in the computer 
literature to the definition of measures of system performance and 
effectiveness . I 1 " 1 "] These definitions have become further complicated 
with the availability of large-scale multiple-access computer systems. 
In this paper, it is sufficient for us to assume that the user has 
determined his system requirements together with the associated system 
constraints. It is hoped that this paper will provide a framework to 
allow the EDP system evaluator to combine the selected relevant system 
performance measures and the related cost elements to arrive aL a 
rational defendable selection decision. 

The approach adopted for selection itself must represent a trade- 
off between effort (time and resources) applied and credibility achieved, 
There are some analysts who feel that throwing a multi-sided die may be 
sufficient to select among qualified vendors. Such a process certainly 
reduces the expenditure of selection resources, but unfortunately it 
suffers in the areas of repeatability, defendability , credibility, and 
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acceptance by the competing vendors and the selection authority who 
is responsible for the final decision, 

A number of EDP equipment selection procedures have been described 
in the literature U7-25J an( j a f ar greater number have undoubtedly been 
used but not formally reported. None of the reported procedures have, 
in the opinion of the authors of this report, satisfactorily handled 
the problem of combining performance and cost. In the last analysis, 
all methods must make use of an explicit determination of the worth* 
to the user of the variety of features proposed by the competing ven- 
dors. Such methods depend heavily upon intuitive judgment. It is the 
identification of rhese judgment areas and the degree to which they 
can be rendered explicit and defendable that contribute to the "success 11 
of a particular selection process. 

In this report we have attempted to apply cost -effectiveness 
analysis techniques to the problem of EDP equipment selection. The 
term cost-effectiveness has been much maligned in the literature and 
in certain political circles in the past few months. It is not our 
purpose to enter into the socio-political aspects of these techniques, 
but rather to apply, to the problem of EDP system selection, methods 
and techniques which have proven superior in the concept formulation 
and systems planning phase of the systems acquisition process. 

It should also be pointed out that the cost-effectiveness analysis 
techniques developed in this report specifically for application to 
EDP system selection are also applicable to the source selection process 
in other system areas. 

V 






A more detailed discussion of worth will be presented later in the 
report, in Section III. ' 

* * 
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SECTION II 
OVERVIEW OF THE SELECTION PROCESS 



In general, the evaluation and selection process involves three 
main components as indicated in Figure 1 ; 

1. Statement of User Needs 

2. Submission of Vendor Proposed System 

3. Measurement or Comparison of the Proposal 
Against the Stated User Needs 

Each of these parts will be discussed in more detail and various 
terms will be defined for later reference. 



USER NEEDS 

The first task to be performed is to determine and make explicit 
an approved set of user needs which will form the basis of the Request 
For Proposal (RFP) and the evaluation and selection procedure. For an 
EDP system, the user needs can be expressed mainly by the description 
of the future workload which the user feels he will have to process 
during the operational life of the EDP system. The main difficulty in 
expressing these needs is that while the user may be able to express 
accurately his current workload, he is never really sure about the 
future workload. It is very difficult for the user to predict what 
he may be asked to process as much as five or more years after the RFP 
has been issued. 

While user needs are expressed mainly in terms of EDP jobs, there 
are other needs or constraints which relate to the EDP system's ability 
to perform these jobs. One such constraint is the maximum time allowed 
to perform any one job or a total set of job*. For example, the user 
may feel that : 

(a) he needs a two-second response time for some task; 

(b) some set of jobs must be completed during an eight- 
hour first shift operation; 

(c) the monthly workload must be completed in less than, 
say, 600 hours ; 

(d) the system must be delivered within 90 days after 
contract award. 
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Figure I. Overview Of The Selection Process 
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It is important to realize that some of these constraints may be 
quite firm to the user; others may be "open-ended 11 , i.e. are really 
"desires' 1 . Several factors complicate the problem of clearly stating 
user needs. These factors are: 

(a) the uncertainty of the future workload; 

(b) the lack of a cost limitation recognized by the 
user (his main pressure may be to satisfy the future 
workload he feels he will be called upon to meet 
independently of cost); 

(c) the lack of cost-sensitivity information regarding 
what it costs to meet different combinations of 
user needs. 

Because of these difficulties, some compromises must be made 
between the user and the external agencies in the procurement chain, 
such as AFADA and ESQ, to arrive at an approved set of user needs 
which will be used as the basis of the Source Selection Plan. 

PROPOSED SYSTEM 

Upon receipt of the RFP the vendor performs various cost -performance 
trade-offs to configure an EDP system which in the vendor's opinion will 
"best" meet the stated user needs. 

This Vendor System consists of: 

(a) hardware having stated technical characteristics; 

(b) software including the various programs required 
to support operating systems, compilers, etc.; and 

(c) vendor support including required maintenance, 
documentation, training of user staff, and systems 
analysis • 

However, there are other parts of the Total System which the user 
must provide to make the system function. This User System includes 
the user's operators, analysts, programmers and facilities. 

MEASUREMENT 

The Total System as proposed by each vendor must be compared 
against the stated user needs, and the winning vendor selected according 
to specified criteria. There are two parts to such an evaluation: 






A system performance evaluation which determines 
the effectiveness of the system. Here effective- 
ness is defined* as the degree to which the system 
will meet the future workload and satisfy the 
constraints. 

A cost evaluation which derermlnes the total cost 
for procurement, operations and maintenance in 
performing the future workload over the total 
required operating life of the syscem. 



.). 



Implicit in the evaluation is the need co validate the vendor's 
proposal. It should be stated that this requirement is common to 
all evaluation procedures and, from a technical point of view, may 
represent the most time-consuming part of the evaluation. This area 
will be discussed in more detail below under the subject of Vendor 
Uncertainties. \ 



• 



SELECTION OBJECTIVE 

* 

To compare alternative selection procedures, an explicit defini- . 
t ion of the selection objective is required. Ihe objective selected 
is as follows: 






"To select a proposed EDP System which performs ■' * I 
a set of future EDP jdbs and meets the job con- 
straints at the Lowest Total Cost to the Air Forcfe, 
* taking into account Job Uncertainties and Vendor 
Uncertainties." 

This objective includes the following three major concepts: 

1. All vendors must show that their proposed systems 
can perform the future EDP jobs and meet the job 
constraints. 

2. Lowest total cost to the Air Force should be the 
selection criterion. 

3. Vendor and job uncertainties are the key factors 
which make the evaluation selection process a 
difficult: one. Hence let us now discuss the 
various aspects of these uncertainties. 



it 

This term will be discussed in greater detail in Section III. 
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UNCERTAINTIES 

As discussed above, the selection process is complicated by two 
classes of uncertainties - vendor uncertainties and job uncertainties. 
Each of these classes will now be discussed in greater detail. 

Vendor Uncertainties 

The proposal submitted by the vendor, in addition to cost and 
contractual -type information, will include the following technical 
information. 

Technical Characteristics 

These are the specifications of the components of the 
proposed computer configuration together with detailed information 
about the performance of each (e.g. speed, capacity, etc.). Assuming 
that the equipment will be delivered on time, one can question whether 
each component will perform at the levels claimed. 

Software 

In response to the RFP, the vendor will describe those 
software packages that he will make available with his equipment. 
Again, assuming that the packages will be available when needed, one 
can question what elements and functions are provided, how well each 
is implemented, and how well each may be used. 

System Performance 

Not only must the elements of hardware and software be 
, considered individually, but their interrelationships must also be 
considered in determining their effects on system performance. For 
example, a card reader or a printer may not be able to run at rated 
speeds because of other system requirements, or a software package 
may degrade system performance because it produces inefficient code, 
because it may constrain an operating program by reducing the amount 
of storage space available, or because it is not suitably matched to 
the available hardware. 

S uppor t 

As mentioned above, one must always be concerned with 
the ability of the vendor to deliver his equipment and associated 
software as scheduled. This is just one example of a number of 
vendor-dependent activities that can be grouped together under the 



heading of vendor support. For example, the reliability of the 
vendor-supplied equipment and programs can very strongly affect 
estimates of system timing. Also, the user's ability to operate 
the vendor's equipment will depend on the documentation available 
and on the professional capability of the analysts and support 
personnel provided by the vendor. Finally, it must be realized 
that both equipment and software must be maintained. The vendor's 
ability to do this efficiently and systematically will also influence 
the user's ability to attain predicted system performance. 

Coping With Vendor Uncertainties 

A number of techniques have been developed for dealing with 
vendor uncertainties. Basically the requirement that the vendor 
supply off-the-shelf equipment and undergo a live-test demo istration 
removes a large part of the risk associated with making state-of-the- 
art systems operational. Of course, this procedure has a compensating 
drawback in that it may prevent the user from acquiring newly devel- 
oped systems. 



The available techniques may be categorized into three major 
areas: 

Professional Personnel 

The basic ingredient for any evaluation is the avail- 
ability of competent professional personnel. Such personnel must be 
carefully trained to stay abreast with the state-of-the-art not just 
in the equipment alone, but also in the way this equipment may be 
used such as through time-sharing or in computer complexes. The 
ability of these people to interpret and assess vendor claims will 
be further enhanced through experience. In particular by working 
with vendors, a better understanding can be acquired of the features 
of the vendor's equipment and staff as well as of the marketing 
strategies employed by the various vendors. In addition, through 
contact with various Air Force user installations, a better under- 
standing can be acquired of the user needs and problems against which 
the vendor proposals must be assessed. 

Tools 

The problem of validating vendor proposals can be greatly 
facilitated by having the proper set of tools. Within the inventory 
of applicable tools, one can identify the following categories: 
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(1) Simulation Programs . 1 ' J Basically it is desirable 

to have a program that can take as input a description of the job to 
be performed together with the specifications of the equipment proposed. 
The output of the program would be an analysis of system performance 
(e.g. overall problem timing, buffered times, component times, storage 
requirements, etc.). Such a program can serve as a check on the ven- 
dor's logic, analysis, and calculations. By incorporating into the 
program an independently derived data base for the equipment specifi- 
cations, such a program provides a cheese on the vendor's data accuracy. 
Simulation techniques are being extended to evaluate some of the dynamic 
aspects of multiprogramming/multiprocessing systems. 

(2) Benchmark Programs . The most satisfactory way to 
validate a vendor proposal is to run an actual live test. Because of 
the time and cost involved in testing the whole job, one is led to make 
use of a program or set of programs that are representative of the job 
to be performed and constitute a predetermined fraction of the total 
job. Such programs can be selected to test the performance of indi- 
vidual computer components as well as to measure, through suitable 
extension factors, the overall system performance. Even though it is 
difficult to design such benchmark programs to test all of the signif- 
icant aspects of the vendor's proposal, nevertheless the use of such 
programs tends to restrain the vendor and to encourage his use of more 
defendable estimates. In general, the benchmark programs are selected 
as portions of the actual expected workload, but programs already devel- 
oped for other jobs or artificially designed can be used provided they 
are suitably representative and can be extrapolated to give the infor- 
mation desired. 

(3) Software Test Programs . An important class of bench- 
mark programs is one especially designed for testing software. Here 

it is more efficient to design programs to test specific elements or 
combinations of elements of the software package. However, a job- 
oriented program may still be useful to test such features as compiler 
efficiency. There is a modest amount of cooperative effort being 
expended under the direction of USASI (U.S.A. Standards Institute) to 
develop such compliance test programs for COBOL. 

(4) EDP Data Base . Information concerning the avail- 
ability and performance of EDP equipment can be organized into a data 
base to facilitate the validation of vendor proposals. Not only does 
this provide a reservoir of information to check figures against, but 
it also serves as a repository for cataloguing acquired experience 
with vendor equipment and claims. As time allows, one can envisage 

a sort of "Good Housekeeping 11 approach to test the performance of 






computer components and software packages with the test results being 
incorporated into the data base, 

(5) Analysis/Synthesis . In support of any validation 
procedure, there must exist a basic understanding of the performance 
of the individual computer components and the interrelations that 
govern how these components work together as part of the overall 
computer system. For example, the performance of the CPU, storage 
devices, I/O devices, data channels, file structures, scheduling 
disciplines, etc., must be analyzed in order to predict the overall 
system performance or to determine those elements that may be criti- 
cal co that performance. Such prediction and estimation must take 
into consideration dynamic conditions that characterize the on-line 
use of computers today. 

Systematic Procedures 

Because of the large number of parameters that contribute 
to the overall complexity of validating vendor proposals, systematic 
procedures must be established to provide an orderly context for 
assessing the vendor's proposal. Given a competent, professional 
staff with an appropriate set of tools, it is still necessary to 
establish an unambiguous set of procedures to assure that the vendor 
understands the user's requirements, and that the evaluators under- 
stand each vendor's proposal. The user's requirements can be formal- 
ized into a set of system specifications which can be translated with 
the cooperation of the evaluation team into the Request For Proposal 
(RFP) that is transmitted to the vendor. By carefully establishing 
the format and contents of the RFP, the vendor will know what to expect 
and what to look for in the RFP. By establishing lines of communica- 
tion between the vendor and the user/evaluation team, the vendor can 
inform the team of critical areas in his proposal and can receive clari- 
fication of any questions on the RFP that may arise. By following sys- 
tematic procedures, one can assure that relevant information is equif-abl 1 
disseminated to all competing vendors. Records can be maintained to 
determine what information was exchanged in case of misunderstandings 
thcit may later arise. By applying established validation procedures 
ami evaluation techniques, one can increase the probability that the 
vendors will accept the results of the validation and evaluation exer- 
cises. In addition, as new techniques or improvements are developed, 
they can be more readily incorporated into the established procedures. 
Finally, by having an established chain of approvals for the selection 
plan and decisions, one has available a set of checks and balances that 
will assure the vendor of equitable treatment and avoid the aura of 
mistrust which might otherwise becloud the vendor/ evaluator relation- 
ship. 









10 



Job Uncertainties 

As was discussed above in Section II, a number of factors 
contribute to the difficulty of explicitly stating the user's needs. 
For example, given that the user will be asked to perform a certain 
job in the future, a number of aspects of that job may change in the 
future and be difficult to predict at the present time. The size of 
the job may vary due to changes in the lengths of the files to be 
processed (e.g. the number of fields in a record or the number of 
records might change). The frequency of running certain jobs may be 
difficult to predict and consequently the total time demanded for 
that job becomes uncertain. Complexity of jobs may increase through 
the incorporation of additional processing steps into the job as 
experience and requirements evolve, or through the introduction of 
more refined or sophisticated methodology. Finally, theset of jobs 
to be performed may change by the addition or substitution of new 
jobs that were not anticipated when the user originally specified his 
needs . 

The level of credibility of the user's predictions of future 
workload can be raised by applying more time and resources to the 
analysis of the user's requirements. For example, if a particular 
selection involved a specified, approved workload to che exclusion of 
unanticipated future jobs, then through the use of detailed systems 
analysis and extensive system simulation, one could very accurately 
establish the characteristics of even a complex workload. However, 
in most cases there is insufficient time or manpower or budget to 
permit the extensive analysis required for accurate workload predic- 
tion. In addition, user jobs do change over time to a degree which 
may be difficult to predict. 

Coping With Job Uncertainties 

Recognizing that the future workload cannot be considered 
fixed and completely specified, evaluators have devised a number of 
techniques to cope with these uncertainties. The most commonly 
adopted method makes use of a "point -scoring" procedure that estab- 
lishes a hierarchy of factors or criteria together with appropriate 
formulas and weights . 123,28] Points are then allocated in accordance 
with how well each vendor has scored on the various factors and upon 
the relative weights allocated by the evaluation team to these factors, 
The vendor with the largest total score is then adjudged to be the 
winner. A detailed presentation of such a procedure for selecting 
among alternatives has been carried out by Dr. James R. Miller. i^\ 
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The remainder of this report will be concerned with the 
application of cost-effectiveness analysis techniques to the problem 
of EDP equipment selection. It is felt that the adoption of a cost- 
effectiveness approach provides a more rational and equitable basis 
for comparing competing vendor proposals and for coping with the 
uncertain workload. Further remarks on the advantages of the cost- 
ef fectiveness approach will be made in Section IV, 

i 

V ' ' * 
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SECTION III 
COST-EFFECTIVENESS ANALYSIS APPROACH 



With the previous statement of the problem and the various 
evaluation difficulties as background, we shall now discuss the 
approach taken in applying conventional cost-effectiveness analysis 
to this problem of computer evaluation and selection, 

DEFINITIONS 

Let us start the discussion by defining various terms which 
will be applied in the analysis:* 

1. Effectiveness : The degree to which a system will perform 
the future jobs and satisfy the constraints. Effectiveness is 
generally considered to consist of the following three main compo- 
nents as illustrated in Figure 2: 

a * Capability : The degree to which a system will perform 
the future jobs and satisfy the constraints, assuming that the system 
is always available for operation and will never malfunction. Capa- 
bility can be measured in various ways, but the two key measures of 
capability are: 

(1) Quality of the work output. This measure is in 
general multi -dimensional since it encompasses 
the many sub-measures used to measure the work 
output. For example, it might include the 
straightness of a line of print or the maximum 
number of copies of printout. 



(2) Time . Given that the quality of the work output 
can be measured, a second key measure of the EDP 
system capability is the time taken to perform 
the future workload, again assuming the system 
is available for operation and never malfunctions, 
For example, the measure could be the expected 
time to perform a given monthly workload. 



These concepts are an application of those used in the evaluation 
of weapon systems, Reference is made to such reports as "Weapon 
System Effectiveness Industry Advisory Committee (WSEIAC) , Final 
Summary Report, AFSC-TR-6 5-6, January 1965. 
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Figure 2. Cost-Effectiveness Definitions 
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b. Availability may be defined as the probability that the 
system will be ready for operation when called upon. 

c. Dependability may be defined as the probability of the 
system completing the job satisfactorily, given that it was avail- 
able. 

However, when evaluating EDP systems in which the primary 
measure of capability is the expected time to perform a given work- 
load (given that the quality of the system meets a certain level of 
acceptability), both availability and dependability can correspon- 
dingly be measured in units of non-productive time expected during 
the performance of a given (monthly) workload. This non-productive 
time consists of two sources: 

(1) Down time: both scheduled maintenance to prevent 
malfunctions and unscheduled maintenance to detect 
and correct malfunction. 

(2) Lost time: non-productive run time requiring rerun 
because of suspected or detected errors. 

Availability also includes how well the vendors can meet the 
desired delivery date, since this has an impact on the non-productive 
time of the system. 

2. Cost : The total dollars required to procure, operate, and 
maintain the system to perform the future set of EDP jobs. As indi- 
cated previously in Figure 1, all costs are to be included in making 
this computation for both the vendor and the user/ 



THE SELECTION PROBLEM 

Given that the effectiveness and cost of a particular system 
can each be measured separately, the evaluation team will still be 
faced with the ptoblem of how to combine these two factors to reach 
a final selection. For example, as illustrated in Figure 3, we might 
have a situation where System B provides a higher level of effective- 
ness than System A but costs more. The source selection problem 
is, "Which is the better system to buy? 1 ' This could be restated as, 



In some cases certain non-differentiating costs may be omitted. 

Note that the decision is straightforward if we have a "dominant" 
case where one system provides more effectiveness at a lower cost. 
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M Is the additional amount of effectiveness worth the added amount of 
cost? 11 It is impossible to answer this question, except on a purely 
intuitive basis (which may be wrong or difficult to defend), without 
resorting to either of the two source selection criteria used in a 
cost-effectiveness analysis : [31] 

1. Specify a level of effectiveness which all systems must 
meet, and select that system which meets this level at 
lowest total cost. This criterion is called "Pivoting 
on Constant Effectiveness 11 « Thus, if E2 is chosen as 
the comparison level of effectiveness and the effective- 
ness of System A is increased accordingly, its new "Opera- 
ting Point" on Figure 3 might be either at A^ (lower 

cost than B and hence selected), or A^ (higher cost 
than B and hence rejected), 

2. Specify a level of cost which all systems must not ex- 
ceed, and select that system which provides the highest 
level of effectiveness. This is called "Pivoting on 
Constant Cost", Thus, if C2 is chosen as the comparison 
level of cost, and the cost of System A is increased 
accordingly, its new "operating point" on Figure 3 might 
be either at A2 (higher effectiveness than B and hence 
selected) or A^ (lower effectiveness than B and hence 
rejected) . 

The first selection criterion will be used for illustration for 
the remainder of this report, since, in general, government procure- 
ment uses this criterion. 



PROPOSED SELECTION PROCESS 

We shall now discuss in greater derail the three categories 
which we have used to describe user needs, a method for quantita- 
tively communicating these needs to the vendors, and a procedure 
for evaluating how well each vendor's proposed system meets each 
of these three categories of needs. 

Classification of User Needs 

Taking into account the many jobs which could make up a 
future workload and the various uncertainties associated with each 
job and with the vendors 1 proposals, efforts were made to apply 
cost-effectiveness analysis methods and techniques to the source 
selection process of Figure 1, The analytical approach used was 
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concentrated on making explicit all of the characteristics of the 
possible jobs and on quantifying the uncertainties associated with 
each job. 

It Loon became apparent that this might not be practical to 
do on every source selection since some might require an excessive 
expenditure of time and user/analyst manpower. Hence it was decided 
to restructure the user needs portion of the selection process as 
shown in Figure 4. User needs can be considered to be made up of 
three primary parts. The first part describes the representative 
workload. The second and third parts which were formerly encom- 
passed by the term "Constraints 11 will be more explicitly defined as 
mandatory requirements and desirable features. 

a. Representative Workload 

Out of the many jobs which the user predicts may make up 
the future workload, only the most important or a selected subset 
would be used to represent this future workload by adjusting job types 
and their frequencies of operation. Jobs again consist of "Known Jobs 11 , 
for which the user has a high degree of confidence in their occurring 
and in their characteristics, such as size and frequency of operation, 
and "Likely Jobs 11 , for which the degree of confidence is lower. An 
analytical technique for expressing this uncertainty by making use of 
quantitative probabilistic estimates will be described in Section 
III. 

b. Mandatory Requirements 

These are absolute requirements which must be met if a 
system is to be considered for further evaluation. A system that does 
not satisfy one or more of these requirements is considered non-responsive 
and is essentially disqualified. Since this is such a strong constraint, 
every effort will be made to keep these requirements to a minimum. 

c. Desirable Features 

This third category is required to express user needs 
because of the following reasons: 

(1) As indicated previously, the representative workload 
only approximates the actual expected workload. Since there may be 
other requirements of the workload that will not be measured in the 
system timing determination, inclusion of selected desirable features 
allows the user to account for the additional capability provided by 
these features to satisfy the above additional requirements. In a sense, 
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these features offer the user a "hedge 11 against uncertainty in his 
statement *of the expected workload. 

(2) Since all measurement techniques have some uncer- 
tainty connected with them, the inclusion of a list of desirable 
features also serves as a hedge against this uncertainty in measuring 
a vendor : s ability to meet the future workload. 

(3) Many times a user need which is called "mandatory" 
l* really only a "desire". For example, is a two-second response 
time for a task absolutely mandatory, or would 2,1 seconds be per- 
missible for a system which is 20% less expensive.' it is better to 
list non discriminatory "design goals" to the vendor in this category 
rather than in mandatory requirements, since elimination of a vendor 
foe slight non-responsiveness (at large decreases in cost) may not 
really be defendable. 



(4) As long as the production of computing equipment 
continues to be competitive with the introduction by different vendors 
of improved design features, the user is forced to consider the benefits 
to be obtained from these features. If the availability of these fea- 
tures wlII be of benefit to the user and if those benefits are not ade- 
quately incorporated in the system timing determination, some means 
should be provided in the selection process to account for them. 

The Representative Workload 
a . D escription 

We shall now discuss a method for explicitly describing 
the representative workload. This description will be used by the 
vendors in preparing their system proposals and will be used as a 
b*asi.T for evaluating the performance of each vendor. 

In explicitly describing the representative workload, 
the analyst must deal with the uncertainties that may exist for each 
pol(.* o( time in the future. This can be treated as two basic prob- 
len^ first, there is the estimation of the user's workload for a 
particular future time including a quantification of the uncertain- 
ties in this workload expressed in probabilistic form; secondly, 
there is the estimation of how the workload may change with time in 
the future which may be based on extrapolations of current and past 
workload data. 

(1) P robabilistic Workload Description . To express this 
uncertainty analytically, the user is asked to define for a given point 
in tune, a reference workload and to provide a quantitative estimate of 
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the probability P that the actual workload occurring at this time may 
exceed the specified reference. This can be expressed by selecting 
certain multiples of the reference workload and having the user specify 
the probability that the actual workload at the selected time will be 
equal to or exceed each multiple of the reference workload. For example, 
in Figure 5 the user has stated that for a particular time there is a 
probability P^ equal to 1.0 that the future workload will exceed the 
reference workload, and a probability P *y equal to 0.80 that it will 
exceed 1.1 times the reference workload, etc. Several comments can be 
made about such an estimate: 

(a) These are the user's subjective estimates and will have to 
be justified to higher level authorities such as AFADA and 
Hq. USAF. 

(b) While theoretically there may be some small probability that 
the actual workload will exceed the upper bound shown, e.g. 
2.0 times the reference workload, the probability can be taken 
as zero if it can be mutually agreed that this will be taken as 
the practical upper design limit for the EDP system. 

(c) While we have described a situation where a workload may vary 
in size, it may also vary in complexity or in any fashion 
which the user chooses to make explicit and include in his 
estimate. The probabilistic description presented above can 
bo used for each element of such a workload specification. 
The workload discussed below will consist, where appropriate, 
of combinations of such elements. 

(J) The units for expressing workload will depend upon the 

particular situation in question. In general, some common 
measure such as equivalent machine time will be used. 

(e) Since the user has provided the analyst in the example illus- 
trated in Figure 5 with four estimates of workload levels, the 
entire workload range may be divided into four segments of in- 
terest (viz. Sj, S^* S«, S,) as shown in the figure. To obtain 
estimates for intermediate workload levels, the original four 
estimates have been connected by straight lines. If greater 
estimation accuracy is desired, additional estimates would 
have to be provided by the user. 
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(2) Workload Growth With Time . As indicated previously, 
workloads change and generally grow with time. Hence a probabilistic 
estimate of the representative workload is needed for various periods 
of time. The summation of this data may be structured as a function 
of operational year as shown in Figure 6a. In the example shown, a 
total operational life of five years and a linear growth of workload 
with time are assumed. Obviously, this linear workload over time is 
only an approximation to the real situation expected which may" vary 
irregularly due to seasonal or irregular demands. However, even this 
approximation to the actual demand function can serve as a design 
guide and evaluation measure. 

In Figure 6a, each workload line has been assigned 
a probability level corresponding to the levels selected in Figure 5. 
For example, the lowest line in Figure 6a represents a workload as a 
function of time which the user has indicated has a 100% probability 
of being experienced or exceeded. In other words, the user has speci- 
fied that he is completely certain that his workload will be at least 
as great as the amount shown by this lowest line. 

The lowest line in Figure 6a has also been labeled 
as the "Reference Workload 11 . The specification by the user of his 
reference workload is independent of the probability level he attaches 
to that level. In other words, the user will first specify his ref- 
erence workload by whatever predictive cools he has available. Then, 
through an independent operation, he will assign a probabilistic level 
to chat workload. 

An even coarser approximation to che actual workload 
is shown in probabilistic form in Figure 6b. Here the average work- 
load for the year is used for each encire year, so that the workload 
increases in discrete amounts. As will be seen later, such an approxi- 
mation simplifies the calculations. 

b. Calculation of Total Expected System Cost 

By constructing an explicit, demand function, i.e. the 
probabilistic workload, the user has staced the range of possible 
workloads for which he is concerned. Every vendor must show, by 
various means open to him, that his EDP system can meet any workload 
up to the indicated maximum which the user may require. These means 
may include equipment expansion or replacement at a later time. It 
may also include the use of service bureau leasing or satellite opera- 
tion if this is acceptable to the user. 
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Since the vendor will be provided with the user's esti- 
mates of the predicted workload, he may now perform various cost- 
performance trade-off analyses resulting in a proposal of an initial 
system installation together with system growth when and if the actual 
workload reaches certain levels. The vendor costs for the proposed 
initial system and its growth will also be provided in the proposal. 

Based on this information, the expected total cost of 
meeting the probabilistic workload can be calculated. The example 
shown in Figure 7 illustrates the hypothetical response of Vendor A 
to the workload described in Figures 5 and 6. This vendor has pro- 
posed to install initially a system A^ which can perform the. workload 
for the first two years of operation within a stated mandatory require- 
ment of less than 600 hours (allowing the remaining time for scheduled 
and unscheduled maintenance as well as any necessary reruns of errors). 
However, during subsequent years, the probable increase in workload may 
exceed the 600 hours available on system A^. In fact, based on the 
stated workload, there is a 100% chance that this will occur in year 5, 
if it did not occur sooner. To cope with this increase, the vendor 
has proposed that his initial system A^ be altered through addition or 
replacement to a new system, called system A2, which can perform the 
increased workload within the 600 hour limitation and which will be 
available when the user so directs. The validated timings for each 
system to perform the different workload levels are shown in Figure 7. 
The vendor also provides cost information indicating his proposed costs 
for all elements of each system, i.e. A^ and A 2> as a function of system 
running time. Such cost information would include shift costs if rele- 
vant as well as lease vs. buy information. 

To these costs which the vendor would provide, the cost 
analyst would add the costs which the user would incur in operating 
the system over the total operational life. Based on this total cost 
data, the total expected cost C for operating each vendor's proposed 
system can be calculated for each year from the formula 

C = p^ + p 2 C 2 + + p n C n (1) 

where p^ is the probability that the actual workload will be con- 
tained in the segment S^ and incur a total cost C^ , and n is 
the total number of segments used to represent the workload range for 
that year. These segments can be determined by the analyst based on 
the user's description of his workload. 

In the example we have selected, the probability that the 
actual workload will fall within any one segment S^ is found simply 
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[32] 
by taking the difference between the two cumulative 1 J probabilities 

that bound that segment. For example, the probability that the work- 
load will fall within the segment S^ is the difference between the 
probability that the workload will equal or exceed the reference work- 
load (1.0) and the probability that the workload will equal or exceed 
1.1 times the reference workload. Referring to Figure 5, the prob- 
ability that the workload will fall within segment S^ is p^ » 0.20 
(viz. 1.00 - 0.80). Similarly, we could determine the probability p„ 
that the workload will fall within each of the other segments S^ . 

These probabilities p^ (i ■ 1, ..., 4 in this example) 
can then be used in equation (1) to determine the total expected cost 
for that operational year. The determination of the cost C^ to be 
applied to each segment will depend upon the amount of information 
available to the analyst and the accuracy that the analyst requires 
in his calculation. For example, referring to Figure 7 for vendor 
system A^, there is a probability of 0.20 that the actual workload 
will fall within segment S^ ; i.e. between 260 hours and 320 hours. 
If we can assume that the probability is distributed uniformly within 
this range of workload and that the cost is proportional to the work- 
load, then we can determine C^ as the cost for system A^ to perform 
an> average workload of 290 hours. 

Similarly, it can be seen from Figure 7 that there is 
a probability p 2 = 0.30 that the actual workload will fall within 
segment S2 , between 320 hours and 370 hours, for vendor system A^. 
Again, assuming it is permissible to use the average workload, cost C2 
will be determined for vendor system A^ to perform the average work- 
load of 345 hours. In a similar fashion, probabilities P3 and p^ 
together with costs C3 and C4 can be determined. Equation (1) 
could now be used to determine the expected cost of operation for the 
first year, viz. P\^\ + P2^2 + ?3^3 + ?4^4 ' ^ e same procedure could 
then be applied to each operational year to determine the expected cost 
for that year. 

Several comments regarding this method should be made: 

(1) System discontinuities may occur inside a segment. 
For example, the vendor may indicate a shift from system A^ to A2 
in S]i , the third segment of the third operational year. Hence to 
calculate properly the expected cost of the third year f s operation, 
a separate calculation for each of the two sub-segments must be made. 
In the above example, the probability that the workload will lie 
within each sub-segment would be determined from a further analysis 
of the cumulative distribution function of Figure 5 reproduced again 
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in Figure 8. If an assumption is made that the cumulative distribu- 
tion function of Figure 8 consists of straight line segments which 
connect the estimates provided by the user, then linear interpolation 
inav be employed. For example, the probability that the actual work- 
load is between 1,0 and 1,05 times the representative workload is 
0,10. A similar breakpoint or sub-segment would be used to represent 
other discontinuities that may occur in system costs such as might 
accompany shift changes, Again linear interpolation could be used to 
determine the probability associated with each of the resultant sub- 
segments . 

{2) Non-productive time, previously discussed under 
system availability and dependability, can be handled :n sever*! 
ways. The method previously described assumed that the reliability 
of all vendor systems cannot be differentiated from one another, 
since they will all meet certain minimum standards. Hence an arbi~ 
trary maximum productive time, e.g. 600 hours, may be chosen for all 
vendjrf , and the vendor system expansions could all be based on this 
upper limit. On the other hand, the evaluators could permit each 
vendor to calculate his maximum productive time, and use this figure 
for expansion design purposes. Obviously, this method wil 1 require 
validation of vendor claims, but this could be done by using vendor - 
claimed reliability as part of the contract, if the vendor would 
agree to do so. The burden of proof for such validation is scill 
on the vendor. 

(3) Note that total system time was used to describe 
each workload for which the corresponding co3t element C^ was 
determined. In an actual case, the time corresponding to the uti- 
lization of each equipment component in the proposed configuration 
would be determined depending upon the vendor's cost elements. 
Again, depending upon the information available and the accuracy 
desired, the analyst could introduce simplifying assumptions to 
keep the calculations tractable and commensurate with the evalua- 
tion modr;l selected. 

(4) After the expected cost, for performing each year ! s 
work. >ad is calculated, we must still combine this "Cost Stream K 
over time into one total expected cost. This can be done using the 
standard approach of obtaining the total present worth by reflecting 
each of these costs back to time zero using an appropriate interest 
rate. In the same fashion, the lease vs. buy calculation may be 
performed to determine which present worth is the lower cost. 
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c . Benefits 

Let us now indicate some of the benefits of the pro- 
posed method of specifying the workload in probabilistic form as 
contrasted with a deterministic method of specifying workload. The 
deterministic procedure requires that the user provide one estimate 
of the representative workload for any given time rather than a 
"band*' of estimates, as in the probabilistic approach. Thus the 
uncertainty is bidden rather than being made explicit. Under that 
procedure, a user is forced to insert some factor of safety in 
making his estimate which may be unduly high, since there are pres- 
sures on him to provide service to his users. 

Providing only one estimate of workload to the vendor 
in the RFP does not permit him to perform suitable cost-performance 
trade-off analysis, since the vendor is not given any information 
indicating the worth of excess system capacity to the user. Pro* 
viding the vendor with a range of values permits him to see the 
upper limit that has been set, as well as the estimated likelihood 
of reaching different workload levels. It thus permits the vendor 
to design a system capable of expanding to meet possible future 
growth requirements and to determine the worth of 3uch an evolu- 
tionary system design in terms of the costs and expectation of 
using these growth increments. In this way the vendor can more 
effectively evaluate his alternative system configurations prior 
to submitting his proposal. This may reduce the number of alter- 
native proposals which a vendor submits. 

Using the proposed approach the source selection team 
can evaluate the vendor proposal in terms of its total expected 
cost. By including considerations of growth and determining their 
cose implications rather than asking the vendor if growth is avail- 
able but not costing ic, a more accurate estimate of the total cost 
of e^ch vendor's proposed system can be obtained. 

Mandatory Requirements 

As discussed above in Section III. with reference to 
Figure 4, a second part of the selection process is the satisfying 
of the mandatory requirements. Each system can be readily evaluated 
against the mandatory requirements since, by definition, all systems 
must meet these or the vendor is considered non -responsive. For 
this reason, when the source selection plan is constructed, the list 
of mandatory requirements should be limited to those characteristics 
which can be firmly defended on a "go/no-go" basis. Any feature 
which the user desires, but cannot firmly defend, should be cate- 
gorized as a desirable feature. 
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Desirable Features 

As discussed previously, the evaluation team must also 
consider a set of desirable features as a hedge against uncertainty 
in the user's statement of his expected workload, as a hedge againsc 
the evaluator' s uncertainty in measuring the vendor's capabilities, 
and as a source of additional vendor capability that was not ade- 
quately covered in the system timing. 

a. Problem Statement 

There are several reasons why the problem of evaluating 
desirable features is a much more difficult one than handling the 
first two elements of user needs, i.e., representative workload and 
mandatory requirements. First, it is almost certain that each of the 
vendors will submit a different "mix 11 of desirable features, ranging 
from none at all to all of the features requested. However, even 
though two vendors submit the same feature, each may have a different 
level associated with it, e.g., one billion versus two billion char- 
acters of IAS. Thus, the first problem is how to quantitatively 
measure the effectiveness of the combination of desirable features 
which each vendor offers. The evaluator can do this by attempting 
to determine the benefits to the user jobs which each feature con- 
tributes and then taking into account the interrelationship of sev- 
eral features as they contribute jointly to the accomplishment of 
user jobs. In developing a method for evaluating a particular fea- 
ture, a way must be found to relate the characteristics of that 
feature to the jobs whose performance will be benefited by it. In 
general the direct effect of a system feature is felt in the time 
(system and/or staff) or quality^of performing the jobs. If it can 
be determined that the effects of a feature have been adequately 
covered in the estimates of system timing previously calculated, 
then that feature need not be considered separately in the list of 
desirables. If this is not the case, then specific steps must be 
established for evaluating the feature. 

Even if the evaluator could solve the first problem 
of evaluating the benefits contributed by each desirable feature, 
the evaluator still has the problem of determining if the difference 
in effectiveness among vendors is worth the difference in cost. 
Figure 9 illustrates in simplified form this problem of source 
selection with respect to desirable features. Consider two vendors 
each of whom performs the future workload at the same expected cost. 
However, assume that vendor A has proposed a minimal EDP system 
containing none of the desirable features listed in the RFP, but 
that vendor B has provided one of the desirable features as part of 
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his proposal at a cost AC greater than vendor A's. Assuming char 
the performance of both machines is identical in all respects except, 
for this desirable feature, we could state qualitatively that the 
effectiveness of vendor B's system is greater than vendor A's. How- 
ever, is the increased effectiveness worth the additional cost of 
AC ? 

As indicated previously, the fundamental principle being 
employed in the evaluation is to pivot on some constant level of effec- 
tiveness and to choose the vendor who provides this level of effective* 
ness at lowest cost. Unfortunately, the inclusion of desirable features 
makes it difficult to define a constant level of effectiveness. The 
user has stated that, in addition to accomplishing a certain workload, 
he desires that additional features also be provided. The solution to 
this problem can be found in the realization that if the user desices 
these features for meaningful reasons, then he must expect to have 
certain jobs which will benefit from these features. However, since 
the user has not made the provision of these features a mandatory re- 
quirement, he is implying that there must be alternative ways of accom- 
plishing these jobs if the desirable feature is not available. This 
information enables the analyst to choose the proper level of effective- 
ness for selection purposes. This will be the level of satisfactorily 
performing the entire approved set of user jobs in accordance with 
approved standards of performance. Since each desirable feature con- 
tributes to some of these jobs, it now becomes a question of comparing 
the proposed cost of any desirable feature against the cost of other 
alternatives which can be used to do the same job(s). This approach 
to system selection translates the task of evaluating desirable features 
to one of cost analysis. It also leads to the concept of the "worth 1 ' 
of a desirable feature in doing a job. 

The term M worth !l has been subjected to diverse economic 
interpretation, For our purposes we will define the "worth" of a 
feature in doing a job as the lowest incremental cost to do the same 
job if the feature is not available. If the vendor's cost is less 
than the user's f, worth n , then that: feature will be acquired from the 
vendor and the cost will be added to the total system cost. If the 
vendor's cost exceeds the user's "worth 11 or if the vendor does not 
provide the feature, then the user will make use of an alternative 
way and add the corresponding cost to the vendor's total system cost. 
If the vendor's cost for the feature is not separately identifiable, 
then there is no way to determine if his cost exceeds the user's 
"worth" and his proposed cost will not be cnanged. 

This process for evaluating desirable features will be 
illustrated by an example in Section III. A key element in this 
process is the determination of the "worth" of a desirable feature. 
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b. Methods of Evaluating Worth 

We can distinguish between two basic ways for deter* 
mining the "worth 11 of the desirable feature. One method makes use 
of analysis, the other makes use of comparative ranking. Each of 
these methods will now be eKamined. 

(1) Analytical Determination of Worth . Since the 
worth of a feature in doing a job has been defined as the lowest 
Incremental cost to do the same job if the feature is not avail- 
able, then to determine the "worth 11 , one must first identify the 
alternative ways of doing the job if the feature were not avail- 
aole. As indicated diagrammatically in Figure 10, there may be 
several ways of doing the job without using the feature. The cost 
of each alternative way should be determined, and then, making use 
of the. probability that the associated job will be performed, the. 
total expected cost over the operational life of the system should 
be determined for each of these alternative ways. The worth of the 
feature wlII be the least of these costs. 

In deciding on whether to utilize a particular 
desirable feature, the evaluator must also consider the cost of 
alternative ways of obtaining the feature and doing the job using 
the feature. For example, he might buy the feature directly from 
another source, Alternatively, as for example in the case of a 
software feature, the user might develop the feature using his 
OvJn resources (in-house). If possible, then, the evaluator would 
make his decision based on doing the job(s) for which the desired 
feature is intended at lowest- total cost. This might be called 
his ''efficient' 1 solution. 

Thus the efficient solution may be chosen by de- 
termining the lowest cost method of obtaining the feature (and 
doing the job), comparing this cost against the worth, and choosing 
the lowest cost alternative. This process is illustrated in Figure 
1 1 * This figure indicates diagrammatically the cost-effectiveness 
of two proposals both of which perform the same basic workload and 
satisfy the mandatory requirements. These two proposals are assumed 
to be identical in all respects, differing only in that one provides 
a desirable feature F at a total system cost of C, whereas the other 
does not . It is immaterial to the present discussion whether these 
proposals come from the same or different vendors. 

While the cost axis of Figure 11 can be quantified 
in a unidimensional scale, the effectiveness axis may involve a num- 
ber of dimensions to represent the elements of effectiveness. How- 
ever since we are pivoting on the constant level of effectiveness 
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specified by the analyst/user as previously discussed, we can repre- 
sent this level diagrammatically as shown in Figure 11. This means 
that we are insisting that those user requirements, which would bene- 
fit from the availability of the feature, be satisfied by some other 
means if the vendor does not provide feature F. Analyzing the alter- 
native ways available to the user results in alternatives A* and A 2 
with their respective costs as shown. Thus, by our definition, the 
worth of the feature is the incremental cost of providing A^ (that is 
Ci - C^) , 

However, it should be noted that once the worth 
of a feature is determined by analyzing the cost of a number of dif- 
ferent ways of getting the job done if the feature is not used at 
all, this worth must be compared with the cost of all alternative 
ways of obtaining the feature and doing the job using this feature. 
Assume in the example that the same feature is available from one 
other source, labeled F , and that the job could be done with this 
feature at a total system cost of C , as shown in Figure 11. The 
proposed strategy is to choose the lowest cost method of getting the 
job done, whether by acquiring the feature or using a lower cost al- 
ternative. Hence in this case, feature F would be purchased. 

(2) Determination of Worth by Comparative Ranking . 
Sometimes, because of time and manpower limitations, it may not be 
possible to determine the worth of all desirable features by analy- 
sis and considered judgment. In such cases, intuitive judgment can 
be used as a part of the quantitative evaluation. Such an approach 
would be implemented as follows: 

(a) Rank all of the desirable features in order 
of importance. The ranking should be sup- 
ported by deliberation utilizing whatever 
quantitative analysis and considered judg- 
ment may be available. 

(b) Allocate points to each feature, establishing 
its relative worth. Such relative worth "ill 
be based on the rationale developed. 

(c) Translate points to dollars of worth. This 
is accomplished by calibrating one or more of 
the features through determining its worth on 
some analytical basis as previously described. 
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(d) Review the results obtained for intuitive 

soundness which is the only real test of this 
procedure. If the final results of dollar 
worth of each feature do not agree with the 
intuitive feelings of the evaluators or source 
selection plan reviewers, an iteration of the 
previous three steps should be performed, 
focusing on the following tv*o potential sources 
of error: First, should the relative worth 
(i.e., points assigned) be changed? Second, 
are there ways of obtaining the features in 
question at a lower cost than the worth assigned 
to the feature? Obviously the more features 
that are calibrated by analysis, the more accu- 
rate will be this procedure. 

c. Other Evaluation Alternatives 

The most credible way to evaluate a desirable feature is 
tc design a live-test demonstration that will include the effects of 
chat feature. In this way, the results of the test will provide an 
explicit quantitative measure of the benefits of that feature. If 
these results can be incorporated into the overall system timing, then 
the particular feature need not be given any further separate considj* 
eration. If this is not the case, then the results of the test may De 
used in an evaluation by worth as described above. 

However, one of the practical constraints in an actual 
source selection is the cost and effort expended in the live-test 
demonstrations. This means that the number of tests must be kept at 
a minimum with each test designed to serve as many testing functions 
as possible. 

Under special circumstances, the following two evaluation 
alternatives may be justified: 

(1) Establish Design Goals . The user may wish to estab 
list a certain level of hardware or software performance that is char- 
acteristic of the present generation of equipment. If it is difficult 
to express the system requirements or to design the live-test demon- 
stration in such a way as to rule out the proposal of equipment con- 
sidered by the user to be substandard, then it may be desirable and 
justifiable to specify the feature as a non-discriminating standard or 
'"design goal" which all qualified vendors can be expected to meet. For 
example, specifying the level of performance of a card reader, card 
punch, or printer might be justified in this way. 
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It should be noted that if this requirement is dis- 
criminatory among the competing vendors, it would be necessary to 
support this requirement more carefully in terms of system require- 
ments. 

(2) Qualitative Evaluation , If the worth of a desirable 
feature cannot be evaluated quantitatively by any of the above tech- 
niques, a qualitative evaluation should be made and documented for con- 
sideration by the Source Selection Authority. Such qualitative factors 
would only be considered and used as "tie-breakers 11 if several vendors 
were sufficiently close based on the quantitative evaluation. 

d. Examples of Evaluating Desirable Features 

The following examples are offered to illustrate how 
desirable features might be evaluated. It should be emphasized that 
these examples are only representative. Actually such features must 
be considered in the context of the user's system requirements. 

(1) Example 1: Additional Core Storage . The user may 
feel that additional jobs not included in the representative workload 
may occur which will require additional core storage over and above 
that provided by the vendor in meeting the basic workload. While the 
best way of measuring this feature would be to include a job requiring 
large amounts of storage in one of the benchmark tests, it may not 
always be possible to do this. Hence, the evaluator must explore ways 
of performing the job if the additional core storage were not available. 
One way of doing this would be to segment the job into smaller parts. 
Now, what will this do to system costs? First, programmer time will 
increase due to the additional programming load. Second, the system 
running time will increase due to the lower efficiency of the operation. 
Both of these will lead to increased machine and staff time. The size 
of the increase will depend on the complexity of the job and the fre- 
quency of its operation. If no other alternatives were available, the 
cost of segmentation would be the worth of this desirable feature. 

The evaluator must also consider alternative nays of 
obtaining the feature. For example, it may be possible to contract the 
jobs requiring additional core storage to a Service Bureau or some satel- 
lite operation equipped to handle it, if this is satisfactory to the user. 
In this case, the resulting costs would be estimated and the least cost 
of all alternative ways to obtain additional core storage would be deter- 
mined . 

(2) Example 2: Software Feature . If the job(s) needing 
the feature has been included in the live-test demonstrations, evaluation 
of the feature is implicitly included in the system timing obtained, and 
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another evaluation is not needed. If the feature is not included 
in the live-test, non-availability of the feature will most likely 
affect the programmer hours required to develop and maintain the 
systenfs programs. Programmer hours would be affected since the 
programmer would now have to do additional programming Co make up 
for not having the software feature at all. One way to handle this 
would be for the analyst to determine the total number of programming 
hours which would be required to do the programming if the software 
feature were available at some standard reference level. Based on 
this reference level, one can define the term "programmer performance 
factor 11 in the following fashion: 

•' t, t* Reference Time 

Programmer Performance Factor * — t~7 

& Proposed Time 

where the Reference Time is the total estimated programmer time (in 
manhours) taken to program a particular job using this reference level 
of software capability for the feature in question, and Proposed Time 
is the total estimated programmer time taken using the vendor proposed 
level of that feature. Using either live-test demonstration results 
or the collective considered judgment of the evaluators in estimating 
the capability of a software feature to perform a certain class or 
classes of jobs, the evaluation function illustrated in Figure 12 can 
be constructed. Making use of this evaluation function and the total 
programmer hours estimated by the analyst to be required if the soft- 
ware feature were available at the reference level, the programmer 
hours required for the proposed level can be determined. For example, 
if the analyst estimates that for the reference level of the software 
feature, the programmer time would be 3000 hours, and if he. determines 
that the proposed feature has an efficiency of 85%, then the estimated 
programmer hours required by the vendor's system is given by: 

■n tt Reference Programmer Hours 

Estimated Programmer Hours = - : r — 5 £ 

Programmer Performance Factor 

3000 Hours 0/ cn „ 

— = 3450 Hours 



.85 

By subtracting the cost of the estimated programmer hours from the 
least costly method of programming the job(s) if the desirable feature 
were not available, the worth of the feature can be determined. 

Alternative ways of obtaining the software feature 
must also be considered. For example, it may be possible for the user 
to develop the feature in-house, requiring additional programmer and 



40 



1 



i 












l.Or- 



PROGRAMMER PERFORMANCE 
FACTOR • 8 



1.00 



.6 - 



.4 - 



,2 - 









.8 


5 






.7 











.5 











— 












- 













POOR 



FAIR 



GOOD 



EXC. 



CO 



LEVELS OF SOFTWARE CAPABILITY 



Figure 12. Software Feature Evaluation Function 



l-l 

JO 



41 






machine time. Alternatively, he may turn to other software sources 
and purchase the feature as a package. The: least cost of these 
alternatives would then be used along with tne worth in the selec* 
t ion process* 

(3) Example 3: Documentation . Again the question can 
be asked, M How much does it cost the user if the user is forced to 
use inadequate documentation as opposed to 'Excellent' documentation?" 
Ihe added cost might be rhe extra time required for readers of the 
documentation as they struggle to understand what the author had in 
mind. Thus, documentation may be evaluated using the same concept, of 
"efficiency*, a^. described previously, and calculating the larger 

number of staff hours required, based on the lDwe.r efficiency factor , 

due to the unavailability of excellent documentation, if possible, 
the analyst might include In this determination of worth -some measure 
of the co-its incurred due to system malfunction that might: occur 
oecause of the inadequate documentation. As before, the analyst 
would also want to consider alternative ways of obtaining eKcellent 
documentation. The. least cost of these alternatives would then be 
used along with the worth in the selection process. 

Note that for this example, the analyst might prefer 
for various reasons (such as economy* to use an alternative evaluation 
procedure. He could handle documentation as a design goal by requiring 
in the R.FP that certain standards of documentation be satisfied. In 
this way the feature becomes a mandatory requirenent. Alternatively, 
the analyst might choose to process vendor differences in documentation 
qualitatively by noting the differences and including the relevant in- 
formation for consideration only as parr of a tie -breaking procedure. 

e. Remarks 

The determination of the worth of a desirable feature by 
constructing an evaluation function which relates the performance bene- 
f i r . ^ of the feature with cost is the most satisfactory way of evaluating 
a desirable feature if its effects cannot be directly included in system 
timing. It should be emphasized that the judgment of experienced per- 
-:-rel will be required to construct the evaluation functions. In fact, 
the accuracy of the analysis is only as good as the experienced judgment 
and substantiating data available. Undoubtedly the most reliable sub- 
stantiation would come from benchmark tests. While errors in judgment 
are nevejr completely avoidable, there are two compensating features to 
rbe ab )ve approach : 

(1) This type of analysis forces the user and evaluator 
to think through and develop the rationale for the need for desirable 
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features and, hence, is superior to a purely intuitive judgment ap- 
proach, since it is more defendable, 

(2) The rationale and evaluation functions developed 
are made explicit and, hence, subject to review by the Source Selec- 
tion Authority, Thus they can be changed if additional information 
is available. 



SELECTION APPROACH 

Previous sections of this report have indicated how to evaluate 
vendor proposals with respect to workload, mandatory requirements, 
and desirable features. This section will expand upon the steps to 
be followed in evaluating the vendor proposals for their desirable 
features and in making a final selection using all of the data gath- 
ered. To illustrate the approach, a simplified example will be used. 

Determining the Worth of Desirable Features 

The Source Selection Plan approved by higher headquarters 
will include a list of all desirable features to be quantitatively 
evaluated, the dollar worth (or evaluation function which describes 
such worth) for each feature, as well as the lowest known cost of 
obtaining the feature separately. An example of the worksheet to be 
used in the evaluation (which can be constructed as an appendix to 
the Selection Plan) is shown in Figure 13. This figure corresponds 
to an example in which there are three desirable features to be con- 
sidered, i.e., F^, F2, and F3, whose user worths and least costs are 
indicated. (In the example shown, each feature is either provided 
completely or not at all. If various levels of a feature could be 
provided, the evaluation function showing worth as a function of 
level provided would be used instead of the single number.) 

Vendor Submits Proposed Casts 

The proposal submitted by each vendor provides the following 
information to the evaluators: 

a. Total proposed cost for entire system. 

b. Cost of each system and expansion capability required 
to meet the probabilistic workload. 

c. Sufficient information to calculate the total expected 
cost of performing the probabilistic workload, 

d. Cost of each separate desirable feature not included as 
part of the basic system. 
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COST ELEMENTS 
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1. 
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2. 


EXPECTED COST TO DO REPRESENTATIVE WORKLOAD 










300K 


305K 


310K 


3. 


COST OF ADDITIONAL JOB BENEFITS: 




10K 


„_ 
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Figure 13. Evaluator's Worksheet 
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Calculating Cost of Representative Workload - ; 

Utilizing the vendor supplied information, the evaluator 
calculates the total expected cost of each vendor's system to perform 
the total representative workload as previously described in Section 
III. These results are then entered into the evaluator 1 s worksheet 
as shown in Figure 13. 

Validation of Mandatory Requirements 

Based on the vendor supplied information, the evaluator must 
validate that the mandatory requirements have been satisfied. 

Calculating Cost of Additional Job Benefits 

The evaluator inserts into the Evaluator 1 s Worksheet all of 
the desirable features which each vendor has proposed and the incre- 
mental costs associated with each of these options. Note that vendor A 
does not provide any of the three features, whereas the cost of F^ and 
F3 are included in the cost of vendor B and vendor C, respectively. 
Based on the cost information of Figure 13, the evaluator can determine 
for each vendor the least costly of the three alternative ways of re- 
ceiving the benefits provided by each of the desirable features. These 
three alternatives are: 

a. Buying the desirable feature from the vendor (at the 
vendor's proposed cost). 

b. Obtaining the desirable feature from another source 
(at the least cost of feature if obtained separately). 

c. Not buying the feature, but using the least costly 
alternative way to provide the benefits (at a cost 
equal to user worth) . 

The lowest additional user cost for obtaining the desirable 
feature (or its equivalent) is shown in Figure 13 as System Cost. 
Note in the example that the user has stated that the worth of F^ is 
$10K, i.e., he can perform the jobs associated with F^ at an expected 
cost of $10K. Since vendor A does not provide this feature, the user 
will be forced to spend $10K in addition to vendor A costs to meet 
those jobs associated with F^. Vendor B includes this feature as part 
of his asic system and has stated that it cannot be removed or priced 
separately. Hence, the user will not have to spend the $10K when using 
vendor B's system. Vendor C can provide F^ at a cost of $15K. Hence, 
the evaluator decides to eliminate this optional feature from vendor C's 
proposal since its cost is higher than its worth to the user, i.e., the 
cost of an alternative method for the user to perform the related jobs. 
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This same approach is followed in determining which of the other desir- 
able features ace to be included in the evaluation. 

Calculating Total Expected Cost 

The total expected cost to the user is rhen calculated by add • 
ing the cost of each desirable feature (or user cost equivalent) to the 
expected cost of performing the probabilistic Representative Workload, 
This total cost, shown in Figure 13, completes tne cost calculation. 



Several interesting observations can be made from analyzing this 
il lustr at ion: 

1. Vendor A had the lowest proposed cost (since he provided no 
desirable features) as well as the lowest cost of performing the repre- 
sentative workload. On the other hand, winning vendor C had the highest 
proposed cost (since he had proposed all three desirable features) and 
the highest cost of performing the representative workload. However, 
neither of these costs is the proper measure for selection. If one 
believes that the user really does have need for the additional capa- 
bility represented by the list of desirable features, and that he will 
have co spend additional funds (i.e., the worth) if a desirable feature 
is not provided, the true criterion of choice must be based on the total 
system costs, There were two reasons why vendor C had the lowest total 
cost in spite of hi* other higher costs. First, he included F3 at no 
additional cost, arid this was worth $10K. Second, he provided F3 for 
$5K and the evaluators estimated its worth to be §20K. 

2. With this approach there are definite advantages to rhe vendor 
to separate as many desirable features as possible from the basic system 
and provide these as optional cost features at a stated price for each 
feature. The reason for this is that if the calculated worth of each 
feature Ls not stated to the vendors (and it should not be since this 
information may dffect the vendor's price), the vendor has no logical 
way of determining whether to propose a desirable feature or not. Hence, 
he is forced to hedge his bets by submitting alternative proposals, **hich 
may increase the vendor's proposal costs and the evaluacor's selection 
cost^, However, with the proposed procedure, the vendor knows that the 
evaluator will only choose those desirable features which have value to 
the user and reject those whose costs are too high. Hence, the vendor 
wlII feel free to offer a "shopping list" of optional desirable features, 
each at a separate price, as part of his proposal, knowing that he can- 
not be penalized by this strategy. 

In the previous illustration, if vendor C had included the high cost 
of Fi as part of his basic system, his total cost would have been higher 
and he would have tied with vendor B. 
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SECTION IV 
CONCLUSIONS 



SYSTEM FEASIBILITY 

The co£t -effectiveness approach described in this paper 
appears feasible and a test project to apply the method to an actual 
selection is being carried out. 

USER IMPLICATIONS 

This evaluation procedure will permit the user to acquire 
a cost-effective EDP system. It should be emphasized, however, that 
additional analysis and data will be required from the user, relating 
system specifications to its expected use, if this procedure is to be 
implemented. Such data will consist of a representative workload 
expressed in probabilistic form, as agreed upon by the user and AFADA, 
a set of defendable mandatory requirements, and a set of desirable 
features together with the worth to the user for having each feature, 
as determined by the user and ESQ. 

VENDOR IMPLICATIONS 

This more useful statement of user needs and the selection 
criteria to be used allows the vendors to construct a better system 
design by performing more and better cost -performance trade-off 
analyses, based on better information. In addition, by permitting 
the vendor to propose optional features, some of which will be se- 
lected by the evaluators on the basis of its cost being less than 
its worth, the vendor may/ not have to propose separate, alternative 
proposals, each containing different combinations of desirable fea- 
tures. 

EVALUATION IMPLICATIONS 

From the evaluator's point of view, the proposed approach is 
more defendable than other approaches examined for several reasons. 
First, it is operationally oriented and, hence, is more rational and 
should be more understandable to reviewers. Second, it avoids com- 
bining cost and performance factors which is always difficult to jus- 
tify, in favor of choosing that system which will satisfy approved 
user needs at lowest total cost to the Air Force. 
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Since it is more explicit and rational than other pro- 
cedures examined, it offers a means of resolving differences of 
opinion regarding the worth of system features. It should be 
stressed that the overall evaluation framework that has been 
developed does not eliminate the need for vendor validation, 

APPROVAL AUTHORITY IMPLICATIONS 

The proposed approach is consistent with OSD practices 
in the field of source selection. In addition, by quantifying 
the uncertainty in workload, the major factor in the evaluation, 
rather than using a single deterministic estimate, increased con- 
fidence in the final selection is obtained. 

ADDITIONAL REMARKS 

This report is intended to describe how cost-effectiveness 
analysis can be applied to EDP system selection. As the reader 
will appreciate, there are many details involved in the system 
selection that cannot be discussed in this report. Such details 
are difficult to discuss in generality since their relevance de- 
pends strongly on the context in which they appear. We feel that 
the approach we have described in this report towards EDP system 
selection will provide a formal framework into which the details 
of a specific selection can be inserted. It is our intenrion, as 
we gain experience in applying the technique, to supplement this 
report with a more detailed account of how specific elements of a 
selection may be processed. At the time of the preparation of this 
report, we have only considered this technique in relation to a 
review of a number of past EDP system selections. The validity of 
the technique can best be established through demonstration. It is 
our plan to accomplish this by applying the technique to a specific 
EDP system selection. 

APPLICABILITY TO OTHER SOURCE SELECTIONS 

One last point of general interest should be made to 
those readers who have an interest in source selection of systems 
other than EDP systems. We have attempted to show that the general 
principles of cost-effectiveness analysis which have been applied 
so often to the concept formulation or systems planning phase of 
the systems acquisition process can also be applied to the source 
selection process, specifically of EDP systems. The same approach 
can also be applied to other systems. In fact, it may be easier 












to apply this to other areas where the measure of effectiveness 
is more easily defined and measured. For this reason, evaluators 
in other system areas should give thought to the possibility of 
applying the cost-effectiveness approach in a form tailored to 
their problem, as was done for the EDF problem. 
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